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Executive Summary 

The current study examined the validity of the selection process used for the National Merit 
Scholarship Program (NMSP) to identify scholarship winners. Namely, this study examined 
whether students who advanced to higher NMSP recognition levels (Commended Students, 
Semifinalists, and various levels of award winners) had higher college performance, as indexed 
by first-year college grades and second-year retention rates. Based on a sample of nearly 
400,000 college students, the results indicated that students who advance to higher NMSP 
recognition levels did earn higher FYGPAs and were more likely to return for their second year 
of college. In sum, these findings provide validity evidence in support of the NMSP selection 
process for identifying students who are most likely to succeed in college and deserving of a 
National Merit Scholarship. 
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Introduction 

Established in 1955, National Merit Scholarship Corporation (NMSC) conducts two national 
scholarship competitions, the National Merit Scholarship Program (NMSP) and the National 
Achievement Scholarship Program (NASP). In general, both of these programs are rigorous 
academic competitions, but in keeping with the focus of the current study, the specific 
goals of the NMSP as stated in the Guide to the National Merit Scholarship Program, 
include the following: 

• identify and honor academically talented U.S. high school students and encourage 
them to pursue rigorous college studies; 

• provide professional services for corporations, company foundations, colleges 
and universities, and other organizations that wish to sponsor scholarships for 
outstanding participants in the competition; 

• promote a broader and deeper respect for learning in general and for exceptionally 
talented individuals in particular; 

• stimulate increased support for the education of scholastically able students; and 

• encourage the pursuit of academic excellence at all levels of education. 

NMSC selects approximately 50,000 students from the more than 1.5 million entrants to 
qualify for recognition in NMSC programs and compete for scholarships. Of these roughly 

50.000 students, about 34,000 are selected to receive Letters of Commendation and about 

16.000 are selected to qualify as Semifinalists in the NMSP 

The Preliminary SAT/National Merit Scholarship Qualifying Test (PSAT/NMSQT®), an 
assessment cosponsored by the College Board and NMSC, serves as the initial screening 
criteria for the two major scholarship programs for high school juniors. Only those students 
who attain a very high score on the PSAT/NMSQT will advance to the Semifinalist level in 
either program. Semifinalists must then meet multiple, additional requirements such as 
submitting a completed Semifinalist application with an essay, a recommendation letter, an 
endorsement from the high school, evidence of demonstrated leadership and extracurricular 
activity, evidence of good academic standing, and confirmation of PSAT/NMSQT performance 
with SAT® scores in order to progress to Finalist standing (McGuire, 2009). The scholarship 
recipients in the competitions are selected from the candidates in the Finalist pool. 

As is the case with any selection process, "relevant evidence and a rationale in support of 
the intended test use" (AERA/APA/NCME, 1999, p. 11) should be furnished, and the NMSP 
is no exception. For example, a recent report by Marini, Mattern, and Shaw (2011) examined 
the relationship between PSAT/NMSQT scores and first-year college grade point average 
(FYGPA). It was shown that even among high-scoring students (i.e., students with PSAT/ 
NMSQT scores of 200 and above), there were statistical differences in FYGPA, and that 
there was a linear, increasing relationship between PSAT/NMSQT scores and FYGPA. These 
findings indicate that students who perform higher on the PSAT/NMSQT also earn higher 
grades in college, thus lending validity to its use as a component of the overall scholarship 
selection process. Because the entire process for selecting the scholarship winners 
involves many different factors and not solely PSAT/NMSQT scores before final decisions 
are made, the current study will build on the research of Marini et al. (2011) by examining 
whether students who advanced to higher NMSP recognition levels (Commended Student, 
Semifinalists, and various levels of award winners) had stronger college performance as 
indexed by first-year college grades and second-year retention rates. 
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There has been some prior research examining whether National Merit Scholarship winners 
selected in high school do indeed perform better in college than those who were considered 
for, but were not ultimately awarded, the scholarship. Studies have compared National 
Merit Scholarship winners to Certificate of Merit recipients (Holland, 1959, 1960) on college 
freshman grades, academic achievement, personality measures, and vocational preference 
measures. A Certificate of Merit recipient refers to Finalists who did not win a National Merit 
Scholarship award. The results showed that the National Merit Scholarship winners tended 
to have higher GPAs than the certificate recipients, but that both groups performed well in 
college. These studies were somewhat inconclusive, however, because they relied on survey 
response data from the winners and response rates differed. In sum, it is important to assess 
the accuracy of the classification of National Merit Scholarship winners, which is the aim of 
the present study. 


Current Study 

The current study examines the college performance, measured by FYGPA and retention 
to the second-year of college, of students who achieved various levels of recognition in 
the NSMP compared to students who received no award. The main research question was 
whether the process used by the NMSP to identify scholarship award winners is valid in terms 
of subsequent college performance. In other words, do those selected as scholarship winners 
in high school perform better in college than those not selected as scholarship winners? It 
is important to examine the validity of a scholarship selection procedure, such as the NMSP 
because winning a National Merit Scholarship is not only associated with prestige but also 
helps to alleviate the financial burden of college costs. Due to the fact that this is a merit-based 
scholarship, it is imperative that the most academically deserving students — those who are 
most likely to succeed in college — are chosen as winners. This would provide credibility and 
empirical evidence in support of the scholarship selection process that is in place. 


Method 

Sample 

The sample examined included three cohorts of first- 
time, first-year students entering college in the fall 
of 2006, 2007, or 2008, which constituted 654,717 
students from 177 colleges and universities across 
the United States (refer to Marini et al., 2011, for 
more details). National Merit Scholarship Corporation 
provided the College Board with the names of 
Commended Students, Semifinalists, Finalists, 
and scholarship recipients for the corresponding 
years, which were matched to College Board data. 

The scholarship awards were classified into two 
groups: "A Award," which is a national award and is 
selected by the National Merit Scholarship Selection 
Committee, and "B, C, or D Award," which are 
corporate or college-sponsored Merit Scholarships. 

Only students with PSAT/NMSQT scores, SAT scores, a self-reported high school GPA 
(HSGPA), a FYGPA, and second-year retention information were included in the analyses, 
resulting in a final sample size of 386,011 students. 


Do those selected as 
scholarship winners 
in high school perform 
better in college than 
those not selected as 
scholarship winners? 
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Within the sample, there were 1 ,053 "A Award" Winners, 4,813 "B, C, or D Award" Winners, 
2,330 Semifinalists, 20,321 Commended Students, and 357494 students not receiving 
recognition. As stated earlier, each year NMSC names about 50,000 students as high scorers 
based on their PSAT/NMSQT Selection Index. Of these high scorers, about 16,000 are named 
Semifinalists, with the remaining 34,000 given the distinction of Commended Student. About 
15,000 of the Semifinalists qualify as Finalists, with about 8,200 receiving Merit Scholarships. Of 
these 8,200 scholarships 2,500 are National Merit Scholarships selected by the National Merit 
Scholarship Selection Committee with an award value of $2,500. It is important to emphasize that 
the sample in the current study does not include all of the students that NMSC serves nationally. 
For example, over three years, 7500 students would win a National Merit Scholarship award worth 
$2,500 (an "A Award"), but only 1,053 were able to be matched in the sample. The student data 
in this study are based upon the higher education data available to the College Board that can be 
linked to NMSC data. Therefore, the number of students in each recognition level is considerably 
lower than the population of students who were recognized by NMSC. 

Measures 

PSAT/NMSQT scores. Official junior year PSAT/NMSQT scores were obtained from College 
Board records. The PSAT/NMSQT consists of three sections — critical reading, mathematics, 
and writing — each scored on a 20- to 80-point scale. The PSAT/NMSQT Selection Index ranges 
from 60 to 240 and is the sum of the individual scores on the three PSAT/NMSQT sections. 

SAT scores. Official SAT scores were obtained from College Board records. The SAT 
comprises three sections — critical reading, mathematics, and writing — each scored on a 
200- to 800-point scale. 

High School GPA (HSGPA). HSGPA was self-reported and obtained from the SAT 
Questionnaire, completed during registration for the SAT. Students' PISGPAs were on a 12-point 
scale ranging from a maximum of A+ (4.33) to a minimum of F (0.00). 

Retention to the second year. Each participating institution supplied second-year retention 
data for their 2006, 2007, and 2008 first-time, first-year students. Students were assigned a 
value of 1 if they returned for their second year and a value of 0 if they did not return. 

First- Year GPA (FYGPA). Participating institutions provided FYGPA for all first-year, full-time 
students. The range, across cohorts and institutions, is 0.00 to 4.27. 


Analysis 

To evaluate the primary research question of whether NMSP recognition level was related 
to subsequent college performance, the mean FYGPA and second-year retention rate were 
computed for the five recognition levels. For FYGPA, ANOVA was used to test for differences 
between the five levels. A significant F value indicates that at least one group's mean differs 
from the others. Additionally, for the ANOVA analysis, effect sizes were calculated using eta 
squared (iq 2 ), which can be related to Cohen's (1998) effect sizes. 1 

It was also of interest to examine whether the percentage of students returning for their 
second year of college varied by NMSP recognition level. While students leave school for many 
reasons (e.g., Ramist, 1981), returning for the second year of college indicates a commitment 
to education and is associated with other positive educational gains (e.g., Baum & Ma, 2007). 

1. For reference, anti 2 of 0.0099 is considered a "small effect," 0.0588 is a "medium effect," and 
0.1379 is a "large effect” (Cohen, 1988). 
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Because retention is a categorical value, the chi-square statistic was used to test for group 
differences. Significant results from these analyses indicated that there was an association 
between recognition level and retention. 


Table 1 

Mean (SD) High School Academic Performance by NMSP Recognition Level 


Measure 

A Award Winner 

Award 

Winners 

Semifinalists 

Commended 

1MO 

Recognition 

PSAT/NMSQT 

223 (7) 

221 (6) 

222 (7) 

207 (5) 

160 (22) 

SAT-CR 

756 (42) 

734 (48) 

739 (52) 

701 (54) 

553 (87) 

SAT-M 

754 (41) 

731 (50) 

730 (52) 

701 (55) 

571 (90) 

SAT-W 

744 (46) 

719 (52) 

720 (56) 

689 (56) 

548 (87) 

HSGPA 

4.19 (0.18) 

4.05 (0.27) 

3.96 (0.35) 

3.94 (0.34) 

3.60 (0.49) 


Table 2 


FYGPA by NMSP Recognition Level 


NMSP Recognition Level 


N 


Mean 


SD 


A Award Winner 


B, C, or D Award Winner 
Semifinalists 


Commended 


No Recognition 


1,053 

4,813 

2,330 

20,321 

357,494 


3.75 

3.59 

3.48 

3.45 

2.98 


0.30 

0.43 

0.51 

0.51 

0.69 


Results 

High School Performance by NMSP Recognition Level 

With regard to the high school academic performance of the five NMSP recognition 
levels, the level of recognition was positively related to PSAT/NMSQT scores, HSGPA, 
and SAT scores (see Table 1). Specifically, the No 
Recognition students had the lowest academic 
performance, followed by Commended students, 

Semifinalists, and "B, C, or D Award" Winners. The 
"A Award" Winners had the highest mean PSAT/ 

NMSQT score, HSGPA, and SAT scores. 

College Performance by NMSP 
Recognition Level 

Overall, there were significant differences in mean 
FYGPA between the NMSP recognition levels with a 
small to medium effect size, F( 4, 386006) = 3792.94, 
p < .001, ti 2 =.038. For example, the No Recognition 
group had a mean FYGPA of 2.98, compared to 3.75 
for the "A Award" Winners (see Table 2). 

The overall Ftest revealed that among all the recognition levels, at least one differed 
significantly from the rest. However, to find out which pairs differed, post-hoc pair-wise 
tests must be performed. As reported in Table 3, the results of the 10 post-hoc pair-wise 
comparisons used to test for differences among each pair of NMSP recognition levels 


The “A Award” Winners 
had the highest mean 
PSAT/NMSQT score, 
HSGPA, and SAT scores. 
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revealed significant differences 2 (p < .05). These findings show that each recognition level's 
mean FYGPA differed significantly from the mean FYGPAs of the other award levels — "A 
Award" Winners had significantly higher FYGPAs than all other recognition levels, and "B, C, 
or D Award" Winners had significantly higher FYGPAs than all nonwinning recognition levels. 


Figure 1 is provided as a graphical representation of the relationship between NSMP 
recognition level and FYGPA to visually demonstrate the positive relationship between the 
two variables. In other words, these results provide evidence for the validity of the NMSP 
selection process in that students who advance to higher levels of recognition earn higher 
grades in college. 


Table 3 

Post-Hoc Tests Comparing NMSP Recognition Level and FYGPA 

Pair-wise Comparison 

Mean 

Difference 

SE 

P 

A Award Winners vs. B, C, or D Award Winners 

0.158 

0.011 

< .001 

A Award Winners vs. Semifinalists 

0.268 

0.014 

< .001 

A Award Winners vs. Commended students 

0.302 

0.010 

< .001 

A Award Winners vs. No Recognition students 

0.773 

0.009 

< .001 

B, C, or D Award Winners vs. Semifinalists 

0.110 

0.012 

< .001 

B, C, or D Award Winners vs. Commended students 

0.144 

0.007 

< .001 

B, C, or D Award Winners vs. No Recognition students 

0.615 

0.006 

< .001 

Semifinalists vs. Commended students 

0.034 

0.011 

0.021 

Semifinalists vs. No Recognition students 

0.505 

0.011 

< .001 

Commended vs. No Recognition students 

0.471 

0.004 

< .001 


Table 4 


Retention to the Second Year by NMSP Recognition Level 


NMSP Recognition Level 


N 


Mean 


SD 


A Award Winner 

B, C, or D Award Winner 

Semifinalists 


Commended 
No Recognition 


1,053 

4,813 

2,330 

20,321 

357,494 


0.98 

0.97 

0.95 

0.95 

0.88 


0.15 

0.17 

0.21 

0.21 

0.32 


Similar to the FYGPA results, second-year retention rates increased as NMSP recognition 
level increased, with a significant association between level and retention for the overall 
sample, x 2 (4, N = 386,011) = 1494.39, p < .001 (see Table 4). For example, 88% of the No 
Recognition group returned for their second year, compared to 95% of the Commended 
group, 95% of the Semifinalists, 97% of the "B, C, or D Award" Winners, and 98% of the 
"A Award" Winners. 


Analogous to the FYGPA results, Figure 2 is provided as a graphical representation of the 
relationship between NMSP recognition level and second-year retention rates to demonstrate 
the positive relationship between the two variables. In other words, these results provide 
evidence for the validity of the NMSP selection process in that students who advance to 
higher recognition levels are more likely to return for their second year of college. 

2. Dunnett's T3 was used to test the multiple comparisons. It is a post-hoc comparison 
procedure in SPSS that is appropriate for comparisons in which the variances and sample 
sizes are unequal. 
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Figure 1 

Mean FYGPA by NMSP recognition level. 
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Figure 2 

Second-year retention rate by NMSP recognition level. 
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Discussion 


In summary, the results indicate that National Merit Scholarship Program recognition levels 
are positively related to both FYGPA and retention to the second year of college. Not only 
do the mean FYGPA and second-year retention rates follow an increasing trend as recognition 
level increases, but there are also statistically significant differences between recognition 
levels, and each difference comparison was associated with a small-to-medium effect size. 


National Merit 
Scholarship Program 
recognition levels 
are positively related 


For the FYGPA analyses, post-hoc pair-wise 
comparisons were computed to test whether any 
two NMSP recognition levels (e.g., No Recognition 
vs. Commended students) had significantly different 
FYGPAs, and all post-hoc pair-wise comparisons 
were statistically significant. This provides statistical 
evidence supporting the NMSP selection process 
in that students who advance to higher NMSP 
recognition levels earn higher FYGPAs and are more 
likely to return for their second year of college than 
their non-award-winning peers. 


to both FYGPA and 
retention to the 
second year of college. 


Similar results have been found in other studies 
examining the impact of receiving a scholarship. 
According to a study done by Yang (2011), students 
in China who received "merit-based aid were 7.4 
times more likely to achieve the top 25 percent major 
rankings." The study also showed that merit aid had 
a larger impact on learning effort and outcomes than 
other types of aid. Henry, Rubenstein, and Bugler 
(2004) compared the cumulative college GPA of 
students who were winners of the Georgia HOPE Scholarship just above the cut off with 
the cumulative GPA of those who did not win the scholarship. They found that a small but 
significant difference in cumulative college GPA existed, and scholarship recipients had, on 
average, GPAs that were .17 points higher than their nonrecipient peers. Coupled with the 
findings from previous research, the findings from the current study provide validity evidence 
in support of NMSC's selection process of identifying students who are most likely to 
succeed in college and most deserving of a National Merit Scholarship. 
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